Local Minimax Learning of Approximately Polynomial Functions

Suppose we have a number of noisy measurements of an unknown real-valued function f near a point of interest x0 ∈ R. Suppose also that nothing can be assumed about the noise distribution, except for zero mean and bounded covariance matrix. We want to estimate f at x0 using a general linear parametric family f(x; a) = a0h0(x) + . . . + aqhq(x), where a ∈ R and hi’s are bounded functions on a neighborhood B of x0, which contains all points of measurement. Typically, B is a Euclidean ball or cube in R (more generally, a ball in an lp-norm). In the case when the hi’s are polynomial functions in (x1, . . . , xd) = x the model is called locally-polynomial. In particular, if the hi’s form a basis of the linear space of polynomials of degree at most two, the model is called locally-quadratic (if the degree is at most three, the model is locally-cubic, etc.). More generally, hi are picked from some linear space of functions H, which must include at least all affine functions on R. Often, information about the behavior of function f on B is available, which is referred to as context, such as, e.g., that f takes values in a known interval, or that it satisfies a Lipschitz condition, etc. Given a loss function and a linear space of functions H, the idea of local minimax learning is in choosing for each point of interest x0 a parameter vector a that minimizes the maximal possible loss over all a ∈ R. The bounds and algorithms are not based on asymptotics or Bayesian assumptions and are truly local for each query, not depending on cross validating estimates at other queries to optimize modeling. The theory of local minimax estimation with context for locally-polynomial models and approximately locally polynomial models has been recently initiated by Jones (2006) and the focus of our paper is on a subclass of problems studied by Jones, which reduce to real algebraic geometry and finite-dimensional optimization of linear functions over compact domains. See Jones (2006) for detailed treatment. Denote by Pr,d the space of real polynomial of degree at most r in d indeterminates. In the case of H = P1,d and a given bound on the change of f on B = {x ∈ R |x| ≤ 1}, the solution for the squared error loss function is in the form of ridge regression, where the ridge parameter is identified; hence, a minimax justification for ridge regression is given, together with explicit best error bounds. The analysis of polynomial models of degree above 1 leads to interesting and difficult questions in real algebraic geometry and non-linear optimization. We show that in the